Music Remixing and Upmixing Using Source Separation

Current research on audio source separation provides tools to estimate the signals contributed by different instruments in polyphonic music mixtures. Such tools can be already incorporated in music production and post-production workflows. In this paper, we describe recent experiments where audio source separation is applied to remixing and upmixing existing mono and stereo music content. 1. AUDIO SOURCE SEPARATION USING DEEP NEURAL NETWORKS Audio source separation algorithms have progressed a long way in recent years, moving on to algorithms that exploit prior information in order to estimate time-frequency masks [1]. For example Deep Neural Networks (DNN), are used in a supervised setting that strongly depends on available training data. In exchange, using supervised training frees them from assumptions needed in other algorithms, such as having recordings from multiple microphones or dealing with repetitive music structures. DNNs are trained to estimate timefrequency masks which still rely on the assumption that energy from different sound sources does not overlap in the time-frequency plane. While applying hard (binary) masks to spectrograms achieves good separation, many noticeable artifacts are introduced. Soft masks produce better sounding results, but imperfect separation. Results from soft masks can still be recombined in remixing and upmixing applications. In this paper we describe two recent prototypes that allow repurposing of musical audio using popular instrument classes. While perceptual evaluation is still pending, both can be used to provide convincing results. 2. REPURPOSING MUSICAL AUDIO The general idea is to use time-frequency masks estimated from DNN models [2] to upmix and remix musical audio. This means that we are able to make audio content interactive by providing the user with controls for remixing or upmixing, not unlike using an intelligent equalizer that knows about the instrument sounds in the mixture. Our prototypes use models trained using the dataset from the SiSEC MUS challenge [3], where sources have been consistently annotated according to common popular music instrument categories (vocals, bass, STFT Mix DNN